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1 Introduction 

This  report  is  a long-range  plan  for  the  NIST  Algorithm  Testing  and  Evaluation  Program  for 
Coordinate  Measuring  Systems  Special  Test  Service  and  is  intended  to  be  reviewed  and  updated 
periodically.  The  audience  is  the  NIST  Calibration  Advisory  Group  (CAG).  The  CAG  was 
established  by  the  NIST  Director  to  provide  broad  oversight  for  the  NIST  Calibration  Program  and 
related  calibration  activities.  The  CAG  advises  the  NIST  Director  and  Operating  Unit  (OU) 
Directors  on  matters  relative  to  the  quality  and  management  of  the  calibration  services.  CAG  also 
develops  and  provides  guidelines  for  new  calibration  services  to  be  offered.  One  of  the  guidelines 
states  that  new  calibration  or  test  service  providers  develop  and  update  long-range  plans.  The 
purpose  of  the  long-range  plan  is  to  outline  the  projected  evolution  of  the  service  so  that  the  service 
evolves  systematically. 

2 ATEP-CMS  Background 

The  NIST  Algorithm  Testing  and  Evaluation  Program  for  Coordinate  Measuring  Systems 
(ATEP-CMS)  evaluates  the  performance  of  data  analysis  software  used  in  coordinate  measuring 
systems  (CMS).  ATEP-CMS  is  the  first  U.S. 

Calibration  Program  test  of  software  used  in 
dimensional  metrology  [7].  Figure  1 shows  a 
high-level  diagram  of  how  ATEP-CMS  works. 

ATEP-CMS  allows  customers  to  submit 
requests  to  have  their  data  analysis  software 
tested  and  evaluated  by  NIST.  NIST  then 
provides  the  customers  with  NIST-generated 
data  sets.  The  customers  produce  fit  results 
from  their  data  analysis  software  using  the 
NIST-generated  data  sets.  NIST  generates  fit 
results  from  the  same  data  sets  using  the  NIST 
Algorithm  Testing  System's  (ATS)  reference 
algorithms  [14,15].  The  two  sets  are  then 
compared.  NIST  then  provides  the  customer 
with  a formal  evaluation  result. 

ATEP-CMS's  objective  is  to  provide 
the  dimensional  metrology  community  with  a 
formal  mechanism  for  testing  and  evaluating  data  analysis  software  in  CMSs.  ATEP-CMS  focuses 
on  three  major  goals:  first,  to  provide  industry  with  a mechanism  for  evaluating  CMS  software; 


Figure  1 NIST's  ATEP-CMS  tests  and  evaluates 
data  analysis  software  for  a given  customer  by 
comparing  fit  results. 


second,  to  reduce  measurement  uncertainties  associated  with  such  software;  and  third,  to  respond 
to  industry's  need  for  an  evaluation  service  that  performs  testing  of  CMS  data  analysis  software. 

CMS  behaviors,  characteristics,  and  limitations  are  not  fully  understood.  Since  the  1 980's 
efforts  have  been  made  to  study  and  evaluate  the  effects  of  CMS  data  analysis  software  on  the  total 
measurement  error  (measurement  uncertainty)  of  a CMS.  Also,  European  efforts  of  the  German 
Physikalisch-Technische  Bundesanstalt  (PTB)  [13]  and  the  British  National  Physical  Laboratory 
(NPL)  [6]  began  to  address  CMS  data  analysis  software  in  the  early  '80s.  Concerns  in  the  confidence 
of  results  reported  by  CMSs  have  been  growing  since. 

• In  1983,  PTB  held  Round  Robin  Tests. 

• In  1983,  The  American  Society  of  Mechanical  Engineers  (ASME)  started  standards  work 
in  coordinate  measurement  machine  software  and  in  1988  held  the  ASME/NSF 
Workshop  on  Mechanical  Tolerancing  [2]. 

• In  1988,  the  Government-Industry  Data  Exchange  Program  (GIDEP)  [10]  issued  an  alert 
on  CMS  software.  The  GIDEP  alert  documented  problems  with  tolerance  computation 
software  from  several  different  vendors. 

• In  1988,  ASME  and  NIST  (formerly  the  National  Bureau  of  Standards  (NBS))  held 
Round  Robin  testing  of  reported  results  from  different  CMM  manufacturers  and 
vendors. 

• In  1989,  DoD  inspectors  temporarily  shut  down  production  lines  due  to  problems  with 
the  inspection  software. 

Throughout  NIST's  involvement  in  addressing  the  problem  of  testing  and  evaluating  software 
found  in  CMSs,  industry  has  voiced  the  need  for  a formal  mechanism  to  test  and  evaluate  CMS  data 
analysis  software.  From  1993  until  now  the  requests  for  NIST  to  provide  a testing  service  for  CMS 
software  have  increased  dramatically. 

3 Requirements  for  ATEP-CMS  - Customer  Demand 

ATEP-CMS  is  intended  to  provide  service  to  the  dimensional  metrology  community. 
Potential  customers  are  CMS  vendors  for  CMMs,  vision  systems,  theodolites,  photogrammetry,  etc. 
Other  intended  customers  are  for  example  CMS  users  in  mechanical  parts,  electronics,  geodesy,  and 
anthropometry.  Calibration  service  providers  are  also  possible  customers.  Currently,  NIST  has  a 
list  of  over  80  companies  that  have  expressed  interest  in  ATEP-CMS.  Some  of  these  companies 
have  already  evaluated  the  ATS;  others  have  been  using  the  ATS  for  internal  testing;  still  others  have 
provided  feedback  for  the  continuing  work.  Most  of  these  companies  are  current  potential  ATEP- 
CMS  customers. 

Providing  ATEP-CMS  is  also  economically  viable.  By  some  estimates,  there  exist  an 
estimated  200,000  CMMs  in  use  worldwide  with  approximately  100,000  installed  in  the  U.S.  There 
are  also  more  than  100  U.S.  CMM  software  producers. 
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4 Alternatives  to  NIST  providing  ATEP-CMS 

Dimensional  measurements  need  to  be  traceable  to  national  standards.  ISO  9000,  DoD,  and 
many  contractual  relations  require  national  standard  traceability.  Currently,  there  are  no  formal 
mechanisms  for  testing  and  evaluating  CMS  data  analysis  software  in  the  United  States.  Therefore, 
there  is  no  traceability  available  for  the  output  of  data  analysis  software.  The  lack  of  this  type  of 
traceability  breaks  the  chain  of  the  dimensional  metrology  standards  that  are  available.  There  ARE 
standards  for  dimensional  metrology  hardware.  There  ARE  NO  standards  for  testing  and  assessing 
the  performance  of  dimensional  metrology  software.  Yet,  the  hardware  that  complies  to  existing 
standards  runs  with  software  that  is  not  traceable  to  any  standard. 

Providing  ATEP-CMS  at  NIST  begins  to  fill  the  existing  gap  in  the  traceability  of 
dimensional  measurements.  NIST  provides  the  dimensional  metrology  community  with  traceability 
to  a national  standard  of  performance  through  the  ATEP-CMS. 

Much  work  has  been  identified  in  the  area  of  testing  CMS  software.  The  International 
Organization  for  Standardization,  under  its  working  group  ISO/TC  3AVG  10  is  working  on  a 
standard  for  software  testing.  NPL  has  prepared  a document  entitled  Proposed  DRAFT  ISO 
Standard:  Method  for  Testing  Software  for  Computing  Gaussian  Substitute  Elements  in  Co-ordinate 
Metrology  [11].  In  addition,  Germany  offers  a service  to  test  CMS  software  comparing  results  for 
test  data  sets  to  results  obtained  from  reference  software  [16].  Also,  Great  Britain  has  proposed 
mechanisms  for  testing  form  assessment  software  [5]. 

There  are  no  domestic  alternatives  to  the  ATEP-CMS  service.  ATEP-CMS  allows  U.S. 
customers  to  domestically  obtain  traceability  of  data  analysis  results  to  an  internationally  recognized 
calibration  body. 

5 Resource  Requirements 

ATEP-CMS  makes  use  of  three  major  components.  First  is  a testing  system,  in  this  case  the 
NIST  ATS.  Second  are  test  procedures  based  on  the  ASME  B89.4.10  draft  national  standard  [1]. 
Third  are  reference  algorithms,  which  are  incorporated  in  the  ATS  and  are  used  to  provide  a baseline 
for  performance  comparison.  The  ATS  runs  on  PC  class  computers.  The  minimum  configuration 
is  as  follows: 

80286  or  higher  CPU 

DOS  Version  3.3  or  later 

Hard  disk  with  at  least  2MB  free  space 

EGA  or  VGA  graphics 

High  density  5 1/4"  floppy  drive  or  3 1/2"  drive 

500  KB  free  memory  (recommend  minimum  of  1MB  installed  memory) 

Optional  equipment: 

80X87  math  coprocessor 
Mouse 
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Resources  for  providing  the  ATEP-CMS  service  include  one  personal  computer  configured 
as  above,  one  person  to  run  the  test,  one  person  to  verify  the  test,  a calibration  manager,  and  office 
space  with  appropriate  workstation  for  personal  computers.  In  addition,  secretarial  services  are 
required. 

6 Planned  Evolution  of  ATEP-CMS 

This  section  addresses  the  planned  evolution  of  ATEP-CMS  in  the  following  three  areas: 
first,  the  technical  scope  of  ATEP-CMS  as  a Special  Test  Service;  second,  the  evolution  of  ATEP- 
CMS  towards  a calibration  service;  and  third,  the  administration  of  ATEP-CMS.  The  projected  time 
frame  for  this  work  has  not  been  scheduled  at  this  time.  Time  schedules  will  be  determined  once 
availability  of  resources  is  determined. 

6.1  Technical  scope  of  service 

NIST  laboratories  are  working  with  industry  (through  standards  committees)  to  establish 
ongoing  programs  to  develop  advanced  dimensional  metrology  algorithms.  The  approach  is  to 
establish  a research  consortium  to  sponsor  and  carry  out  work  identified  by  the  national  research 
agenda  on  dimensional  tolerancing  and  metrology.  NIST  will  participate  in  the  development  of  new 
technology  and  standards  for  statistical  tolerancing  and  tolerancing  of  advanced  geometries. 
Collaboration  between  NIST  laboratories  will  lead  to  specifying  objectives  for  data  analysis 
algorithms.  Work  with  consortium  members  will  lead  to  developing  new  reference  software 
implementations  of  the  algorithms. 

Gears,  threads,  airfoils,  and  other  sculptured  surfaces  are  some  of  the  geometries  that  will  be 
addressed.  Also  the  evaluation  of  other  fitting  criteria  like  minimum  zone,  datum  fits,  and  other 
criteria  used  in  industry  will  be  made  available.  In  addition,  NIST  will  add  other  CMS  software 
functions  to  the  CMS  software  evaluation  process.  Some  of  these  functions  are:  tolerance  zone 
evaluation,  parallelism,  position,  material  condition,  and  other  tolerances  from  ASME  Y14.5  [3,4] 
and/or  ISO  1101  [12].  NIST  will  track  and,  where  appropriate,  incorporate  results  of  ongoing 
research  efforts  like  NIST's  computational  metrology  competence  project  and  the  Consortium  For 
Advanced  Manufacturing  - International’s  Dimensional  Inspection  Techniques  Specifications 
(CAM-I’s  DITS)  effort. 

6.2  Evolution  towards  a calibration  service 

NIST  will  track  the  use  of  the  test  service  results  by  industry  to  evaluate  the  utility  of  ATEP- 
CMS 's  measured  values  and  measurement  uncertainty.  Additional  research  will  be  done  by 
collecting  data  on  process  characteristics  and  studying  the  interaction  of  part  deviations,  coordinate 
sampling  plans,  and  data  analysis  methods.  Using  these  results,  performance  metrics  and  assessment 
methods  for  new  algorithms  will  be  developed.  NIST  will  develop  algorithm  testing  tools  based  on 
these  metrics  and  integrate  the  tools  into  an  algorithm  testing  environment. 

Additionally,  through  a coordinated  research  program,  NIST  will  address  any  shortcomings 
identified  by  industry.  NIST  will  also  work  through  appropriate  ISO  technical  committees  and 
working  groups  to  ensure  ATEP-CMS  methodology  is  incorporated  into  international  standards  for 
traceability  and  ISO  9000  certification. 
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6.3  Administration  of  the  service 

In  the  case  where  ATEP-CMS  requests  increase  dramatically,  NIST  will  work  with  research 
laboratories  at  the  University  of  North  Carolina  Charlotte,  Cornell,  and  others,  DoD  and  industry 
standards  laboratories,  and  other  interested  parties  (determined  by  advertising)  to  establish  test 
services  under  NIST’s  Measurement  Assurance  Programs  (MAP). 

7 Other  Information 

7.1  Customer  Feedback 

NIST  has  maintained  communications  with  the  dimensional  metrology  community  via  the 
ASME  standards  committees.  NIST  also  participates  in  the  ISO  efforts  in  this  area.  In  addition, 
NIST  has  made  presentations  at  annual  American  Society  for  Quality  Control  (ASQC)  [8]  and 
National  Conference  of  Standards  Laboratories  (NCSL)  [9]  conferences.  Through  these  channels, 
NIST  has  obtained  feedback  that  is  crucial  to  the  continuing  efforts  of  developing  methods  for 
testing  CMS  software. 

7.2  Voluntary  Standards 

NIST's  development  and  implementation  of  ATEP-CMS  complement  the  work  of  the 
standards  committees  at  ASME.  In  addition,  NIST  is  working  with  ISO  to  develop  a uniform 
international  approach  to  dimensional  metrology  software  testing. 

7.3  Training 

Training  is  necessary  for  ATEP-CMS  service  customers  for  them  to  understand  and  interpret 
test  results.  Administrative  and  technical  documentation  is  available  for  all  ATEP-CMS  customers 
and  interested  parties.  In  addition,  training  is  necessary  for  other  personnel  that  might  run  the  ATEP- 
CMS  service.  Currently,  there  is  sufficient  documentation  to  be  able  to  follow  the  ATEP-CMS 
administrative  process.  There  are  also  clear,  concise  technical  reports  that  would  allow  an  operator 
to  run  the  ATS,  produce  test  results,  and  provide  an  evaluation.  However,  to  be  able  to  have  expert 
ATEP-CMS  operators,  formal  training  will  be  required  specifically  on  using  the  ATS  and  using 
ATEP-CMS  to  produce  evaluations.  A formal  training  plan  will  be  developed  in  the  near  future. 
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